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(54) Data processing apparatus and data processing methods 



(57) Data processing apparatus includes a data 
processing section (2) and a real-time learning section 
(1 ). The data processing section (2) processes input da- 
ta by a predetermined processing method and outputs 
the processed data. The real-time learning section (1 ) 



controls the process such that the processing method 
is learned in real time and the data processing section 
(2) processes the input data by the learned processing 
method, so that the output data is improved as time 
elapses. 
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[0001] The present invention relates to data processing apparatus and data processing methods, and more partic- 
ularly, to a data processing apparatus and a data processing method in which input data which is processed may be 
5 learned In real time such that the input data is adaptively processed and output data obtained by the processing is 
improved as time elapses. 

[0002] Fig. 1 of the accompanying drawings shows a structure of a previously-proposed data processing apparatus. 
[0003] The input data to be processed is sent to a determination section 91 and processing sections 92-, to 92 M . The 
processing sections 92-, to 92 M process the input data by processing methods different from each other and send to 
10 a selection section 93. 

[0004] On the other hand, the determination section 91 controls the selection section 93 such that one of the outputs 
of the processing sections 92-, to 92 M is selected according to, for example, the amount of a feature extracted from 
the input data. The selection section 93 selects one of the outputs of the processing sections 92-, to 92 M according to 
the control of the determination section 93 and outputs as output data. 

15 [0005] In other words, the processing sections 92-, to 92 M process the input data by the processing methods different 
from each other, the determination section 91 selects the most appropriate method among the processing methods of 
the processing sections 92 1 to 92 M according to the amount of a feature of the input data, and the selection section 
93 selects the processing result of the input data obtained by the selected processing method. 
[0006] Therefore, in the data processing apparatus shown in Fig. 1 , when input data is image data and the processing 

20 sections 92-, to 92 M serve as filters having different tap coefficients for filtering the image data, for example, the deter- 
mination section 91 determines among the processing sections 92-, to 92 M a filter having the tap coefficient which the 
designer of the data processing apparatus thinks most appropriate, according to the amount of a feature of the input 
data. The selection section 93 selects the result obtained when the determined filter filters the input data, as output data. 
[0007] As described above, in the conventional data processing apparatus, the processing result of the filter having 

25 the tap coefficient which the designer of the data processing apparatus thinks most appropriate, among the processing 
sections 92 1 to 92 M is selected. It is, however, not necessarily true that processing the input data by such a tap coefficient 
is appropriate for the input data. 

[0008] In other words, in some cases, filtering with the use of a tap coefficient other than the tap coefficients which 
the processing sections 92-, to 92 M have is more appropriate for the input data, and as a result, better improved output 
30 data may be obtained. 

[0009] One aspect of the present invention provides a data processing apparatus for processing input data and 
outputting the processed data, including a data processing section for processing the input data by a predetermined 
processing method and outputting the processed data, and a real-time learning section for controlling such that the 
processing method is learned in real time and the data processing section processes the input data by the learned 

35 processing method, so that the output data is improved as time elapses. 

[0010] Another aspect of the present invention provides a data processing method for processing input data and 
outputting the processed data, including a data processing step for processing the input data by a predetermined 
processing method and outputting the processed data, and a real-time learning step for controlling such that the 
processing method is learned in real time and the input data is processed by the learned processing method in the 

40 data processing step, so that the output data is improved as time elapses. 

[001 1] Further aspects of the invention provide a program for carrying out the data processing method and a storage 
medium for storing the program. 

[001 2] The invention will now be described by way of example with reference to the accompanying drawings, through- 
out which like parts are referred to by like references, and in which: 
45 [0013] Fig. 1 is a block diagram showing a structure of a previously-proposed data processing apparatus. 

[001 4] Fig. 2 is a block diagram of an example structure of a self-tuning processing circuit according to an embodiment 
of the present invention. 

[0015] Fig. 3A is a flowchart of a processing method used before an algorithm is changed. 
[0016] Fig. 3B is a flowchart of a processing method used after the algorithm is changed. 
50 [0017] Fig. 4A is a flowchart of a processing method used before processing elements are changed. 
[0018] Fig. 4B is a flowchart of a processing method used after the processing elements are changed. 
[0019] Fig. 5 is a view showing input learning data used for learning. 

[0020] Fig. 6 is a block diagram showing an example structure of an NR processing circuit according to an embod- 
iment of the present invention. 
55 [0021] Fig. 7A is a view showing the input data to be processed by the NR processing circuit shown in Fig. 6. 

[0022] Fig. 7B is a view showing the-reliability of the input data to be processed by the NR processing circuit shown 
in Fig. 6. 

[0023] Fig. 8 is a flowchart showing the processing of the NR processing circuit shown in Fig. 6. 



2 

BNSDOCID: <EP 1001371A1_L> 



EP 1 001 371 A1 



[0024] Fig. 9 is a block diagram showing an example structure of an image processing apparatus according to an 
embodiment of the present invention. 

[0025] Fig. 1 0 is a view showing a class tap and a prediction tap to be processed by the image processing apparatus 
shown in Fig. 9. 

[0026] Fig. 11 is a flowchart showing the processing of a data processing section 2 shown in Fig. 9. 

[0027] Fig. 12 is a flowchart showing the processing of a real-time learning section 1 shown in Fig. 9. 

[0028] Fig. 13 is a block diagram showing an example structure of an image processing apparatus according to 

another embodiment of the present invention. 

[0029] Fig. 14 is a block diagram showing an example structure of a learning processing section 53 n shown in Fig. 1 3. 
[0030] Fig. 1 5 is a block diagram showing an example structure of a prediction-coefficient learning section 61 shown 
in Fig. 14. 

[0031] Fig. 16 is a block diagram showing an example structure of a class-classification tuning processing section 
62 shown in Fig. 14. 

[0032] Fig. 17 is a flowchart showing the processing of a real-time learning section 1 shown in Fig. 13. 
[0033] Fig. 18 is a view showing class taps to be processed by the image processing apparatus shown in Fig. 13. 
[0034] Fig. 1 9A is a view of a computer into which a program that executes processing related to the present invention 
is installed. 

[0035] Fig. 1 9B is a view showing an example storage medium in which a program that executes processing related 
to the present invention is stored. 

[0036] Fig 1 9C is a view showing cases in which a program that executes processing related to the present invention 
is distributed to the computer through a satellite and a network. 

[0037] Fig. 20 is a block diagram of a computer into which a program that executes processing related to the present 
invention is installed. 

[0038] Fig. 2 shows an example structure of a self-tuning processing circuit according to an embodiment of the 
present invention. 

[0039] In this self-tuning processing circuit, processing is applied to input data, for example, digital data, subjective 
to the processing, namely, input data to be processed, by the most appropriate processing method, and the result of 
the processing is output as output data. 

[0040] Specifically, the self-tuning processing circuit is formed of a real-time learning section 1 or real-time learning 
means, and a data processing section 2 or data processing means. 

[0041] The real-time learning section 1 receives learning input data used for learning the processing method by 
which the input data to be processed is processed. The real-time learning section 1 controls the data processing section 
2 such that the processing method is learned in real time according to the learning input data and the processing 
method obtained as the result of the learning is used for processing, in order to adaptively process the input data to 
be processed so that the output data is improved as time elapses. 

[0042] The data processing section 2 adaptively processes the input data to be processed by the processing method 

specified by the real-time learning section 1, and outputs the result of the processing as output data. 

[0043] Therefore, since the data processing section 2 processes the input data to be processed by the use of the 

processing method learned in real time as described above, the processing is adaptively applied to the input data to 

be processed. The output data obtained by adaptively processing the input data to be processed is improved as time 

elapses. 

[0044] More specifically, the processing method by which the data processing section 2 processes the input data to 
be processed is improved by real-time learning in the real-time learning section 1. The data processing section 2 
adaptively processes the input data to be processed by the use of the processing method specified by the real-time 
learning section 1 . As time elapses or the amount of input data processed increases, a learning effect gradually appears, 
and the output data output from the data processing section 2 is improved as time elapses. The improvement of the 
output data means that the processing method of the data processing section 2 is improved so as to be appropriate 
for processing the input data to be processed, as time elapses. 

[0045] As described above, in the self-tuning processing circuit, the processing method for the input data to be 
processed is learned in real time and the input data to be processed is processed by the processing method obtained 
as the result of the learning such that the output data is improved as time elapses. In other words, the processing 
appropriate for the input data to be processed is applied to itself as learning proceeds. Therefore, the processing which 
the self-tuning processing circuit shown in Fig. 2 performs can be called self-tuning processing. 
[0046] An improvement of the output data means, for example, an image-quality improvement when the input data 
to be processed is an image. More specifically, it means an improvement of the S/N ratio or resolution. When the 
purpose of the processing in the data processing section 2 is an effect such as smoothing or edge emphasis, the 
improvement of the output data means that the effect greatly appears. Therefore, the improvement of the output data 
means that the output data is improved so that output data having a state which the user desires is obtained. 
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[0047] The real-time learning section 1 learns in real time the processing method used for processing the input data 
to be processed, according to the input learning data input to the section. As shown by dotted lines in Fig. 2, the input 
learning data is the input data to be processed, input to the data processing section 2, the output data, such as the 
frequency characteristics of the input data to be processed, output from the data processing section 2, or further, an 

5 auxiliary input data sent to the self-tuning processing circuit to perform learning of the processing method applied to 
the input data to be processed. Two or more of the above data may be used as input learning data. 
[0048] In the self-tuning processing circuit, the processing method of the data processing section 2 is improved every 
moment when the real-time learning section 1 learns in real time. Therefore, the self-tuning processing circuit is a so- 
called change-with-time system. The improvement of the processing method can be roughly divided into a case in 

io which the algorithm of the processing is changed and a case in which the contents of processing steps constituting 
the series of processing or elements constituting the processing are changed. 

[0049] When "x" indicates the input data to be processed and "y" indicates output data, for example, the processing 
in the data processing section 2 is expressed by an expression of y = f(x) with the use of a function n f( ) u . 
[0050] When the expression is set to n f(x) = a,^ 2 + DqX + c 0 ,° for example, the processing of the data processing 
is section 2 used before an algorithm is changed can be expressed by a flowchart shown in Fig. 3A. 

[0051] In a step S1 , c^x 2 is calculated. In a step S2, b^ is calculated. Then, in a step S3, ^x 2 + b 0 x + c 0 is calculated 
to obtain the output data "v." 

[0052] In this case, changing the form of the function "f(x) H corresponds to changing the algorithm of the processing. 
More specifically, changing the function from "f(x) = a 0 x 2 + b^ + c 0 " to n f(x) = (a^ 2 + b^ + c 0 ) 2 , " for example, corre- 
20 sponds to changing the algorithm of the processing. 

[0053] In this case, the processing of the data processing section 2 after the algorithm is changed can be expressed, 
for example, by a flowchart shown in Fig. 3B. 

[0054] Specifically, in steps S1 to S3, the same processing as that shown in Fig. 3A is executed, and (aoX 2 + b^ + 
c 0 ) 2 is calculated to obtain the output data "y." 
25 [0055] As clearly understood from the comparison between the flowchart shown in Fig. 3A and that shown in Fig. 
3B, when the algorithm of the processing is changed, the flowchart which expresses the processing is also changed 
accordingly. 

[0056] Changing a coefficient of the function "f(x)" corresponds to changing the contents of an element constituting 
the processing. Specifically, for example, changing the function from "f(x) = a^ 2 + box + c 0 n to n f(x) = a-,x 2 + b^x + c/ 
30 corresponds to changing the contents of the elements constituting the processing. 

[0057] In this case, the processing of the data processing section 2 is changed from the processing shown by a 
flowchart in Fig. 4A, which is the same as that shown in Fig. 3A, to, for example, the processing shown by a flowchart 
in Fig. 4B. 

[0058] Specifically, in this case, a^ 2 is calculated in a step SI , b-,x is calculated in a step S2, and then, a^ 2 + ^x 
35 + c 1 is calculated in a step S3 to obtain the output data u y." 

[0059] As clearly understood from the comparison between the flowchart shown in Fig. 4A and that shown in Fig. 

4B, when the coefficients of the function "f(x) M are changed, the contents of steps constituting the processing are 

changed accordingly More specifically, in Figs. 4A and 4B, the contents of the step S1 is changed from the calculation 

of SqX 2 to that of a-,x 2 , the contents of the step S2 is changed from the calculation of box to that of b«,x, the contents 
40 of the step S3 is changed from ^x 2 + box + c 0 to a-,x 2 + b-,x + c 1 . 

[0060] In some cases, the processing method is improved both by changing the algorithm of the processing and by 

changing the contents of an element constituting the processing. 

[0061 ] The functions used in Figs. 3A and 3B and Figs. 4A and 4B are just examples used to describe an improvement 
of the processing method. Whether the output data is actually improved is irrelevant to the fact that the function ex- 

45 pressing the processing of the data processing section 2 is changed as described above. 

[0062] The self-tuning processing circuit differs, for example, from conventional digital filters in that the coefficients 
of the function f(x) are changed by real-time learning according to newly generated adaptive coefficients. Specifically, 
in the conventional digital filters, a tap coefficient may be changed according to input data to a coefficient prepared in 
advance, which is not generated by real-time learning. In other words, in the conventional digital filters, a tap coefficient 

50 which the system designer thinks is appropriate for input data is selected among tap coefficients prepared in advance. 
A tap coefficient adaptive to input data is not obtained. As a result, even if a tap coefficient more appropriate for the 
input data exists, it cannot be used unless it is prepared in advance. Therefore, the above conventional digital filters 
are called in some cases systems changing a tap coefficient H adaptive ly," but accurately speaking, it is not "adaptively" 
but "selectively." They are systems in which a tap coefficient adaptive to input data is not generated but a tap coefficient 

ss selected uniquely against the input data is used. 

[0063] In contrast, in the self-tuning processing circuit, a tap coefficient is generated by real-time learning such that 
input data is adaptively processed. In other words, the input data is processed by the tap coefficient truly most appro- 
priate for the input data. More specifically, for example, whereas in the conventional digital filters, for example, when 
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two sets of tap coefficients are prepared in advance, one of the two sets of tap coefficients which the system designer 
thinks is appropriate for the input data is uniquely selected and used, tap coefficients used for processing the input 
data are improved every moment by real-time learning to obtain tap coefficients more appropriate for the input data, 
and processing is executed with the use of such tap coefficients in the self-tuning processing circuit. Therefore, process- 

5 ing adaptive to the input data is performed. 

[0064] In addition, the self-tuning processing circuit differs from conventional processing which applies tuning 
processing to input data but output data is not improved every moment, such as adaptive dynamic range coding (ADRC) 
processing, in that output data is improved every moment. Specifically, in the ADRC processing, the maximum value 
MAX and the minimum value MIN of the pixel values of pixels constituting a block in an image are detected, and the 

io pixel values of the pixels constituting the block are re-quantized in K bits according to a local dynamic range DR obtained 
by MAX - MIN in the block. More specifically, the minimum value MIN is subtracted from the value of each pixel in the 
block and the difference is divided by DR/2 K for re-quantization. Therefore, tuning processing is applied to input data 
in that the quantization step DR/2 K of the pixel values is obtained according to the block dynamic range DR and the 
pixel values are quantized by the quantization step DR/2 K , but output data obtained as the quantization results is not 

is improved every moment. 

[0065] The real-time learning section 1 uses all the sample values of input learning data obtained so far in order to 
improve the output data as time elapses in real-time learning at the current time "t. " The output data of the data process- 
ing section 2 may be more effectively improved if the real-time learning section 1 gives a weight to sample values 
obtained at positions close in a time manner to the current time T of the input learning data and does not use sample 

20 values obtained at positions far from the current time much. Therefore, the real-time learning section 1 can execute 
real-time learning such that the above-described weight is given to input learning data, namely, the sample values of 
input learning data obtained at positions in a time manner far from the current time T are ignored according to the 
rates corresponding to the degrees of the time differences. Alternatively, it is possible that the real-time learning section 
1 uses only input learning data obtained within a predetermined time period from the current time B t." Specifically; for 

25 example, as shown in Fig. 5, the real-time learning section 1 can use only input learning data within four samples from 
the current time V for real-time learning at the current time w t." This can be implemented by applying a weight of zero 
to input learning data away from the current time "t" by five or more sample periods. 

[0066] In real-time learning at the current time "t, n it is also possible that the real-time learning section uses only 
sample values within a predetermined range from the sample value of the input learning data obtained at the current 

30 time "t." Specifically, for example, in Fig. 5, a sample value such as that of the learning data obtained at the time u t-2" 
has a magnitude largely different from that of the sample value of the input learning data obtained at the current time 
V may adversely affect the learning in the real-time learning section 1 . Therefore, it is possible not to use, namely, to 
give a weight of zero to, a sample value having a magnitude largely different from that of the input learning data obtained 
at the current time n t" in real-time learning at the current time V in the real-time learning section 1. 

35 [0067] Fig. 6 shows an example structure of a noise reduction (NR) processing circuit 100 to which the self -tuning 
processing circuit shown in Fig. 2 is applied, according to an embodiment. 

[0068] When input data having noise is input to this NR processing circuit 100, the NR processing circuit applies 
self -tuning processing to the input data to effectively remove the noise from the input data and outputs output data. 
[0069] Specifically, for example, to simplify a description, a case is examined in which input data having a constant 
40 true value and on which noise fluctuating in time is superposed, as shown in Fig. 7A, is averaged to remove the noise 
fluctuating in time. The noise is effectively removed by setting a weight for input data having a large noise level, namely, 
having a low S/N ratio, to a small value and by setting a weight for input data having a small noise level, namely, having 
a high S/N ratio, to a large value. 

[0070] In the NR processing circuit 100 shown in Fig. 6, as the evaluation value of input data, the degree of reliability 
45 indicating how the input data is close to the true value, which is the reliability of the input data against the true value, 
for example, shown in Fig. 7B, is obtained by real-time learning. The NR processing circuit 100 calculates its average 
wliiie the weight corresponding to the reliability is applied to the input data, to effectively remove the noise. 
[0071] Therefore, since the NR processing circuit 100 shown in Fig. 6 obtains a weighted average of the input data 
by the use of the weight corresponding to its reliability and outputs it as output data. The output data y(t) is obtained 
so from the following expression, where x(t), y(t), and a x(l) indicate the input data, the output data, and the reliability of 
the input data at the time V in the NR processing circuit 100, respectively. 



ss 
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v(t)-i^ (1) 

i-u 



It is assumed here that a larger weight is given to the higher degree of reliability cc^ of input data. 
io [0072] The output data M y(t-1), n obtained one sample before the current time V is calculated as follows from the 
expression (1 ). 
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[0073] As for the output data y(t), the degree of reliability Oy (t) indicating how the output data y(t) is close to the true 
value, which is the reliability of the output data y(t) against the true value, is introduced as an evaluation value of the 
output data y(t). The reliability Oy iu1) of the output data y(t-1 ), obtained one sample before the current time V is defined 
by the following expression. 

t-i 

[0074] I n this case, the output data "y(t)" and its reliability cty (t) are expressed as follows with the use of the expressions 
(1)to(3). 



t-i 



2 a *<i> x (0 + ci. M t)X(t) 

y(o- i ^ 

i-<) 



= — — ( 4 ) 

a y(»-t) +a x(t) 

a y(t) = a y(t-1) +a x(t) 

[0075] The weight used to obtain the output data y(t) at the time T is indicated by w(t), and defined by the following 
expression. 

w (t) = a y(M) /(a y(M) + a x(t) ) (5) 
[0076] From the expression (5), the following expression is satisfied. 
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1 -w(t) = a x(t) /(a y(M) + a x(t) ) (6) 

[0077] With the use of the expressions (5) and (6), the output data y(t) in the expression (4) can be expressed in the 
following way by a weighted average obtained by multiplications and an addition. 

y(t) = w(t)y(t-1) + (1 -w(t))x(t) (7) 

[0078] The weight w(t) (and 1 - w(t)) used in the expression (7) can be obtained from the expression (5) with the use 
of the reliability cxy^ of the output data y(t-1 ) obtained one sample before and the reliability u^t) of the current input 
data x(t). The reliability «y(t) of the current output data y(t) in the expression (4) can also be obtained with the use of 
the reliability Oy (M ) of the output data y(t-1 ) obtained one sample before and the reliability of the current input data 
x(t). 

[0079] When the reliability of the input data x(t) and the reliability Oy (1) of the output data y(t) are expressed, 
respectively, by the reciprocals of the corresponding dispersions a x(t) 2 or o y(t) 2 , in other words, when the reliability 
and the reliability 0Cy (t) are set as follows, 

20 a x(t ) = 1/o \<t) 2 

a y(t) = 1/a y(t) 2 (8) 
25 the weight w(t) in the expression (5) can be expressed by the following expression. 

W ( t ) = 0 x(t) 2/ (< 5 y(t-1) 2 + a x(t) 2 ) (9) 



10 
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[0080] In this case, 1 - w(t) defined by the expression (6) can be expressed by the following expression. 

1 - w ( t ) = ^(H) 2 A^ y0 . 1) 2 + a x(t) 2 ) (10) 
35 [0081] A term, o y(1) 2 , can be expressed by the following expression with the use of the expressions (4), (8), and (9). 

o y( t) 2 = w(t) 2 ^i) 2 + 0 "W(t)) 2 a x(t) 2 (11) 

[0082] The NR processing circuit 100 shown in Fig. 6 learns the weight w(t) in real time according to the expression 
(5) and 1 - w(t) in real time according to the expression (6) with the input data x(t) and the output data y(t) being used 
as input learning data. The NR processing circuit 1 00 calculates the weighted average of the output data y(t-1 ) obtained 
one sample before and the current input data x(t) according to the expression (7) with the use of the weight w(t) obtained 
as a result of real-time learning to adaptively process the input data x(t) so that the noise included in the input data x 
(t) is effectively removed. With these operations, the S/N ratio of the output data y(t) output from the NR processing 
circuit 100 is improved as time elapses. 

[0083] A latch circuit 11 1 receives input data. The latch circuit 11-, latches the received input data, for example, in 
synchronization with the timing when the input data is received, and sends it to a subsequent-stage latch circuit 11 2 
and to an input reliability calculation section 12. The latch circuit 11 2 and a latch circuit 11 3 latch the input data output 
from the previous-stage latch circuits 11-, and 11 2> and send it to the subsequent-stage latch circuits 11 3 and 11 4 , 
respectively, and to the input reliability calculation section 12, in the same way as the latch circuit 11 v The latch circuit ■ 
1 1 4 latches the input data output from the previous-stage latch circuit 1 1 3 and sends it to the input reliability calculation 
section 12. 

[0084] The input reliability calculation section 1 2 receives the same input data as that sent to the latch circuit 11-, in 
addition to the input data latched by the latch circuits 11-, to 11 4 . Therefore, when input data x(t) is sent to the latch 
circuit 1 1 1 and to the input reliability calculation section 12, the input reliability calculation section 12 also receives input 
data x(t-1) to x(t-4) latched by the latch circuits 11-, to 11 4 . The input reliability calculation circuit 12 calculates, for 
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example, the dispersion of input data from the input data x(t) to x(t-4), and sends the reciprocal of the dispersion as 
the reliability (that is, an input reliability) of the input data x(t) to an output reliability calculation section 13 and to 
a weight calculation section 15. 

[0085] The output reliability calculation section 1 3 obtains the reliability oty (t) (that is, an output reliability) of the output 
5 data y(t) according to the expression (4) with the use of the input reliability sent from the input reliability calculation 
section 12 and the output of a latch circuit 14, and outputs it to the latch circuit 14. 

[0086] The latch circuit 14 latches the output reliability oty (t) sent from the output reliability calculation section 13, for 
example, in synchronization with latching of the input data x(t), and sends it to the output reliability calculation section 
13 and the weight calculation section 15. Therefore, the latch circuit 14 sends the output reliability Oy (M) obtained one 
10 sample before, to the output reliability calculation section 1 3 and the weight calculation section 1 5. 

[0087] The weight calculation section 15 obtains a weight w(t) according to the expression (5) with the use of the 
input reliability sent from the input reliability calculation section 12 and the output reliability cey( M ) sent from the 
latch circuit 14, and sends it to a weighting section 16 and an operation section 17. 

[0088] The latch circuits 11-, to 11 4 , the input reliability calculation section 12, the output reliability calculation section 
75 13, the latch circuit 14, and the weight calculation section 15, described above, correspond to the real-time learning 
section shown in Fig. 2. 

[0089] The weighting section 16 multiplies the weight w(t) sent from the weight calculation section 15 by the output 
of a latch circuit 20, and sends the multiplication result to an operation section 19. The operation section 17 subtracts 
the weight w(t) sent from the weight calculation section 15, from one, and sends the subtraction result, 1 - w(t), to a 

20 weighting section 1 8. The weighting section 1 8 receives the input data x(t) as well as the output of the operation section 
17. The weighting section 18 multiplies the input data x(t) by the output of the operation section 17, and sends the 
multiplication result to the operation section 19. The operation section 19 adds the outputs of the weighting sections 
16 and 18, outputs the addition result as output data y(t), and sends it to the latch circuit 20. The latch circuit 20 latches 
the output data of the operation section 19, for example, in synchronization with latching of the input data x(t), and 

25 sends it to the weighting section 16. 

[0090] The weighting section 16, the operation section 17, the weighting section 18 : the operation section 19, and 
the latch circuit 20, described above, correspond to the data processing section 2 shown in Fig. 2, and calculate the 
weighted average according to the expression (7) and outputs it as the output data y(t). 

[0091] The operation of the NR processing circuit 100 shown in Fig. 6 will be described below by referring to a 
30 flowchart shown in Fig. 8. The operation of the NR processing circuit 100 and the control of each functional block is 
achieved by a controller not shown. Alternatively, the entire NR processing algorithm may be implemented by software. 
[0092] First, the input data x(t) is input to the latch circuit 11 1: the input reliability calculation section 12, and the 
weighting section 18 in a step S11. 

[0093] In the next step SI 2, the input reliability calculation section 12 obtains the input reliability a^. 

35 [0094] More specifically, the latch circuit 111 latches the received input data in synchronization with the timing when 
the input data is received, and sends it to the subsequent-stage latch circuit 11 2 and to the input reliability calculation 
section 12. The latch circuit 11 2 and the latch circuit 11 3 latch the input data output from the previous-stage latch circuits 
11-, and 11 2 and send the data to the subsequent-stage latch circuits 11 3 and 11 4 and to the input reliability calculation 
section 12, respectively, in the same way as the latch circuit 11-,. The latch circuit 11 4 latches the input data output 

^0 from the previous-stage latch circuit 11 3 and sends it to the input reliability calculation section 1 2. Therefore, the input 
reliability calculation section 12 receives the input data x(M) to x(t-4) from the latch circuits 11, to 11 4 , respectively, 
at the same time when it receives the input data x(t). The input reliability calculation section 12 uses the input data x 
(t) to x(t-4) to obtain the input reliability cc^ and sends it to the output reliability calculation section 1 3 and to the weight 
calculation section 15, as described above. 

45 [0095] At the timing when the input reliability calculation section 12 sends the input reliability to the weight 
calculation section 1 5, the latch circuit 1 4 latches the output reliability a^.-, > output from the output reliability calculation 
section 1 3 one sample before. In a step S1 3, the weight calculation section 15 uses the input reliability sent from 
the input reliability calculation section 12 and the output reliability oty (M) latched by the latch circuit 14 to obtain the 
weight w(t) according to the expression (5). This weight w(t) is sent to the weighting section 16 and the operation 

50 section 17. 

[0096] The weighting section 16, the operation section 17, the weighting section 18 ; the operation section 19, and 
the latch circuit 20 use the weight (t) output from the weight calculation section 15 to calculate the weighted average 
of the input data x(t) and the output data y(M ) obtained one sample before, according to the expression (7). 
[0097] Specifically, the weighting section 16 multiplies the output of the latch circuit 20 by the weight w(t) sent from 
55 the weight calculation section 15, and sends the multiplication result to the operation section 19. The latch circuit 20 
latches the output data y(t-1) which the operation section 19 outputs the last time, at the timing when the weighting 
calculation section 1 5 outputs the weight w(t). Therefore, the weighting section 1 6 obtains the product w(t)y(M ) of the 
output data y(M) and the weight w(t) and sends it to the operation section 19. 
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[0098] The operation section 1 7 subtracts the weight w(t) sent from the weight calculation section 1 5, from one, and 
sends the subtraction value, 1 -w(t), to the weighting section 18. The weighting section 18 multiplies the output, 1 -w 
(t), of the operation section 17 by the input data x(t) and sends the multiplication result, (1 - w(t))x(t), to the operation 
section 19. 

5 [0099] The operation section 19 adds the output, w(t)y(t-1), of the weighting section 16 and the output, (1 -w(t))x(t), 
of the weighting section 18. In other words, with the use of the weight w(t) output from the weight calculation section 
15, the weighted average of the input data x(t) and the output data y(M) obtained one sample before is calculated 
according to the expression (7). 

[0100] This weighted average is output as the output data y(t) in a step S15. The output data y(t) is sent to the latch 
io circuit 20 and latched. 

[0101] In the next step S16, it is determined by the controller whether the input data to be processed still exists. 
When it is determined in the step S16 that the input data to be processed still exists, the procedure proceeds to a step 
S17 and the output reliability calculation section 13 updates the output reliability. Specifically: the output reliability 
calculation section 1 3 adds the input reliability calculated in the step S1 2 by the input reliability calculation section 
*s 12 and the output reliability <Xy (M) obtained one sample before and latched by the latch circuit 14, according to the 
expression (4) to obtain the current output reliability ay (t)) and sends it to the latch circuit 14. Then, the procedure 
returns to the step S11 , and the same processing is repeated with the next input data. 

[0102] On the other hand, when it is determined by the controller in the step S16 that the input data to be processed 
does not exist, the processing is terminated. 

20 [0103] As described above, the weight w(t) is calculated by the use of the input reliability ot^ of the current input 
data x(t) and the output reliability cxy^ obtained one sample before. Then, according to this weight w(t), the weighted 
average of the current input data x(t) and the output data y(t-1) obtained one sample before is calculated, and output 
as the output data y(t) serving as the processing result of the input data x(t). The output reliability cty (t) of the output 
data y(t) is then calculated by adding the current input reliability to the output reliability Oy (M ) obtained one sample 

25 before. In the same way, the next input data x(t+1), x(t+2), ... is sequentially processed. Therefore, the weight w(t) is 
learned such that portions having high noise levels are not much taken into account and portions having low noise 
levels are sufficiently taken into account in previous input data. In other words, the NR processing circuit 100 obtains 
a weight w(t) adaptive to input data, and as a result, improves the output data every moment as learning of the weight 
w(t) proceeds. The NR processing circuit 100 is configured such that output data obtained by effectively removing 

30 noise from the input data is output. 

[0104] In the above-described case, the reciprocal of the dispersion of the input data, obtained with the use of the 
past four samples and the current input data, is used as the input reliability a^. The input reliability can be, for 
example, the reciprocal of the error square of the current input data against the average of input data, obtained with 
the use of the past four samples and the current input data, or the sum of the reciprocal of the dispersion and that of 

35 the error square. Instead of the past four samples, all input data may be used for calculation. 

[0105] The NR processing circuit 100 shown in Fig. 6 can also be applied, for example : to a case in which noise is 
removed from image data or sound data. 

[0106] In the above embodiment, the NR processing circuit 100 shown in Fig. 6 has been described in the viewpoint 
of noise reduction. The NR processing circuit 100 shown in Fig. 6 has a self-tuning processing nature, in which a 
40 processing method, specifically, a weight coefficient w(t), is learned in real time and the processing is executed by the 
processing method obtained as a result of the learning such that input data is adaptively processed to improve output 
data as time elapses, and therefore the input data is adaptively processed. The NR processing circuit 100 shown in 
Fig. 6 can also achieve, for example, waveform shaping most appropriate for input data 

[0107] In the above case, the NR processing circuit 100 multiplies the output data y(t-1 ) by the weight w(t) and the 
45 input data x(t) by the weight, 1 - w(t), in the expression (7) to obtain the output data y(t). It is also possible that either 
the output data y(t-1 ) or the input data x(t) is multiplied by a weight to obtain an output data y(t). 
[0108] Fig. 9 &hows an example structure of an image processing apparatus 200 to which the self-tuning processing 
circuit shown in Fig. 2 is applied, according to an embodiment. 

[0109] In this image processing apparatus 200, class classification tuning processing is performed to convert an SD 
so (standard density) image to an HD (high density) image. 

[0110] The class classification tuning processing is formed of class classification processing and tuning processing. 
Data is classified into classes according to its structure or nature in the class classification processing, and the tuning 
processing is applied to each class. The tuning processing is performed in the following way. 

[0111] Specifically, in the tuning processing, for example, an SD image is linearly coupled with predetermined pre- 
ss diction coefficients to obtain the prediction values of the pixels of an HD image. With this operation, a high-frequency 
component not included in the SD image is recovered. 

[0112] More specifically, for example, the prediction value E[y] of the pixel value "y" of a pixel, namely, an HD pixel, 
constituting the HD image is obtained by a linear first -order coupling model defined by linear coupling of the pixel 



9 

BNSDOCID: <EP 1 001371 A1_l_> 



EP 1 001 371 A1 



values, namely, learning data, x lf x 2 ... of several SD pixels, that is, pixels constituting the SD image, and the prede- 
termined prediction coefficients w-,, w 2 , .... The prediction value E[y] can be expressed by the following expression. 



E[y] = w lXl + w 2 x 2 + 



(12) 



[0113] To generalize the expression (12), when a matrix "W" formed of a set of prediction coefficients "w," a matrix 
M X U formed of a set of learning data, and a matrix "Y"' formed of a set of prediction values E[y] are defined in the following 
expressions. 
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[0114] The least squares are applied to this observation equation to obtain prediction values E[y] close to the pixel 
values u y" of the HD pixels serving as master data. When a matrix "Y n formed of a set of the true pixel values "y" of 
HD pixels serving as master data and a matrix n E n formed of a set of the remainders "e" of the prediction values E[y] 
against the pixel values "y" of the HD pixels are defined in the following way, 
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the following remainder equation is satisfied from the expression (1 3). 

45 



XW = Y + E (14) 



[0115] In this case, prediction coefficients Wj used to obtain prediction values E[y] close to the pixel values u y u of the 
HD pixels are obtained when the following error square is made minimum. 
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[0116] Therefore, when the result obtained by differentiating the above error square by the prediction coefficient W| 
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is zero, in other words, the prediction coefficient Wj which satisfies the following expression is the most appropriate 
value for obtaining the prediction values E[y] close to the pixel values °y u of the HD pixels. 

de 1 de« 3e m 

e i +e 2 ^ + +e ™ =0(i = 1 ' 2 '-' n) < 15 > 

[0117] When the expression (1 4) is differentiated by the prediction coefficient w h the following expression is satisfied. 

d^ = X »' 3^ 2 = X i2'-- = X in- 0 = 'A-."") (16) 

[0118] From the expressions (15) and (16), the expression (17) is obtained. 

*n m m 

JeiX,! -O.Je^, -0 t -t)c i x ill -0 (17) 

i-l i-L j.l 

[0119] In addition, when the relationships among the learning data u x, n the prediction coefficients n w,' the master 
data "y," and the remainders M e n in the remainder equation (14) are taken into account, the following normal equations 
are obtained from the expression (17). 



(fx il x il )w l+ (|x ll x i2 )w 2+ ... + (|x il x in )w 0 -(^Xiiyi) 

m l m 1 i ~ l 

(J x i2 x u )w L + ( V x i2 x i2 )w 2 f- + (yx i: x ia )w n . ( f x l2Vi ) . g 

i-i i-i frf frf v ' 

™ °L m m 

(2 X i« X il) W l + (2 XinXi2)W2+ * * + (y X inXin)W 0 -(X* x ioyi ) 

i-i w frf frf 



[0120] When the learning data u x u and the master data "y" of appropriate numbers are prepared, the normal equations 
(18) of the same number as that of the prediction coefficients "w" to be obtained can be formulated. Therefore, by 
solving the equations (18), the most appropriate prediction coefficient °w" is obtained. To solve the equations (18), it 
40 is required that a matrix formed of coefficients applied to the prediction coefficients "w" be regular. It is possible to use, 
for example, a sweep off method (Gauss-Jordan deletion method) to solve the equations (18). 

[0121] The most appropriate prediction coefficient u w u is obtained, as described above, and prediction values E[y] 
close to the pixel values y of HD pixels are obtained from the expression (1 2) by the use of the prediction coefficient 
'w° in the tuning processing. 

45 [0122] The tuning processing differs from interpolation processing in that a component not included in the SD image 
but included in the HD image is reproduced. The tuning processing seems to be identical with interpolation processing 
using a so-called into ipolat ion filter if only the expression (12) is viewed. Since the prediction coefficients H w M corre- 
sponding to the tap coefficients of the interpolation filter are obtained by so-called learning with the use of the master 
data "y," a component included in the HD image can be reproduced. In other words, a high -resolution image can be 

so easily obtained. Therefore, it can be said that the tuning processing has a so-called image creation effect, namely, a 
resolution improvement effect. 

[0123] The image processing apparatus 200 shown in Fig. 9 converts an SD image to an HD image by the tuning 
processing described above. 

[0124] In other words, an SD image is input to the image processing apparatus 200 as input data to be processed 
55 and an HD image is input as input learning data. The SD image serving as the input data to be processed can be an 
SD image broadcasted in TV broadcasting. The HD image serving as input learning data can be an HD image to be 
broadcasted together with an SD image through a satellite. In addition, the HD image serving as input learning data 
can be an HD image obtained by applying the above<Jescribed class classification tuning processing to an SD image. 
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It is preferred that the contents of the SD image are identical with those of the HD image. 

[0125] In Fig. 9, a frame memory 21 , a thinning-out circuit 22, a frame memory 23, a class-tap generation circuit 24, 
a prediction-tap generation circuit 25, a class classification circuit 26, an adder 27, a predict ion -tap memory 28, an 
adder 29, a master-data memory 30, and an operation circuit 31 correspond to the real-time learning section 1 shown 
5 in Fig. 2. A frame memory 41 , a class-tap generation circuit 42, a prediction-tap generation circuit 43, a class classifi- 
cation circuit 44, a coefficient RAM 45, and a prediction operation circuit 46 correspond to the data processing section 
2 shown in Fig. 2. An HD image serving as input learning data is input to the frame memory 21 in the real-time learning 
section 1 , and an SD image serving as input data to be processed is input to the frame memory 41 of the data processing 
section 2. 

10 [0126] In the data processing section 2, the frame memory 41 temporarily stores the SD image serving as the input 
data to be processed input to the frame memory 41 , for example, in units of frames. 

[01 27] The class-tap generation circuit 42 extracts as SD pixels disposed around an HD pixel, that is a focused pixel, 
for which a prediction value is to be obtained by tuning processing, for example, a block, that is, a processing block, 
formed of SD pixels disposed within a predetermined distance from the focused pixel, from the SD image, and sends 

*5 it to the class classification circuit 44 as a class tap to be used in class classification processing. 

[0128] In other words, for example, the class -tap generation circuit 42 extracts a class tap formed of four-by-four SD 
pixels indicated by O marks around the focused pixel from the SD image, as shown in Fig. 10, and outputs it to the 
class classification circuit 44 as a class tap for the focused pixel. In the embodiment shown in Fig. 10, the SD image 
has half the number of pixels of the HD image in each of the horizontal and vertical directions. 

20 [0129] The predict ion -tap generation circuit 43 extracts SD pixels to be used for obtaining the prediction value of the 
focused pixel in the prediction operation circuit 46, and sends them to the prediction operation circuit 46 as a prediction 
tap. In other words, for example, the prediction-tap generation circuit 43 extracts the same SD pixels as those extracted 
as the class tap by the class-tap generation circuit 42 from the SD image as a prediction tap, as shown in Fig. 10, and 
sends them to the prediction operation circuit 46. 

25 [0130] In this embodiment, the class tap and the prediction tap are formed of the same SD pixels. The class tap and 
the prediction tap are, however, not necessarily formed of the same SD pixels. 

[0131] The class classification circuit 44 class-classifies the focused pixel according to the class tap sent from the 
class-tap generation circuit 42, and sends the class obtained by the class-classification to the coefficient RAM 45 as 
an address. More specifically, the class classification circuit 44 sets the class of the focused pixel to the value corre- 
30 sponding to the pattern of the pixel values of SD pixels constituting the class tap, for example, a predetermined number 
of SD pixels, and outputs the class to the coefficient RAM 45 as an address. 

[0132] Specifically, the class classification circuit 44 outputs, for example, the value corresponding to the pattern of 
the pixel values of four-by-four, 16, SD pixels around the focused pixel, serving as the class tap, as shown in Fig. 10, 
as the class of the focused pixel. - - - - 

35 [01 33] When the pixel value of each SD pixel is indicated by many bits such as eight bits, the number of the patterns 
of the pixel values of 16 SD pixels is as enormous as (2 8 ) 16 and subsequent quick processing is difficult. 
[0134] As preprocessing of class classification, processing for reducing the number of the bits of each SD pixel 
constituting a class tap, for example, the ADRC processing, can be applied to the class tap. This processing needs to 
reduce the number of bits, and may be quantization processing by the use of a quantization table. 

40 [0135] When the ADRC processing is applied and the value of each pixel constituting the class tap is indicated by 
one bit, the number of the patterns of the pixel values of 16 SD pixels becomes (2 1 ) 16 , which is very much smaller than 
that obtained when the ADRC processing is not performed. 

[0136] The coefficient RAM 45 stores the prediction coefficients of each class obtained when the real-time learning 
section 1 performs real-time learning. When the class classification circuit 44 sends a class, the prediction coefficients 

45 stored at the address corresponding to the class are read and sent to the prediction operation circuit 46. The coefficient 
RAM 45 updates the storage contents every time the real-time learning section 1 sends the prediction coefficients of 
each class. Before the real-time learning section 1 sends the first prediction coefficients to the coefficient RAM 45, the 
coefficient RAM 45 stores the predetermined initial values of the prediction coefficients of each class. 
[0137] The prediction operation circuit 46 uses the prediction coefficients w, w 2 , ... of the class of the focused pixel 

50 sent from the coefficient RAM 45 and the prediction tap x^, x 2 , ... sent from the prediction-tap generation circuit 43 to 
perform the operation indicated by the expression (12). The prediction value E[y] of the focused pixel n y" is obtained 
and output as the pixel value of an HD pixel. 

[0138] On the other hand, in the real-time learning section 1, the frame memory 21 temporarily stores an HD image 
input to the frame memory 21 as input learning data, for example, in units of frames. The thinning-out circuit 22 reads 
55 an HD image which is to serve as master data "y" in prediction -coefficient real-time learning from the frame memory 
21 , thins out pixels from the HD image to change it to an SD image, and sends to the frame memory 23. The frame 
memory 23 temporarily stores the SD image sent from the thinning-out circuit 22. 

[0139] The class-tap generation circuit 24 and the prediction-tap generation circuit 25 extract the class tap and the 
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prediction tap of the focused pixel, respectively, from SD pixels constituting the SD image stored in the frame memory 
23 in the same way as the class-tap generation circuit 42 and the prediction-tap generation circuit 43, and send to the 
class classification circuit 26, the adder 27, and the adder 29. 

[0140] The class classification circuit 26 performs class classification according to the class tap sent from the class- 

5 tap generation circuit 24, obtains the class of the focused pixel, and sends it to the prediction -tap memory 28 and to 
the master-data memory 30 as an address, in the same way as the class classification circuit 44. 
[01 41 ] The adder 27 reads stored values at the address corresponding to the class output from the class classification 
circuit 26, that is, the operation results corresponding to the multipliers of the prediction coefficients °w" in the expres- 
sions (18), from the prediction-tap memory 28, and adds the stored values to the multipliers of the prediction coefficients 

10 n w n calculated from the SD pixels constituting the prediction tap sent from the prediction-tap generation circuit 25 to 
perform the operation corresponding to the summation (L) serving as the multiplier of the prediction coefficient "w" at 
the left side of each of the normal equations (1 8). The adder 27 stores the operation result in the prediction-tap memory 
28 at the address corresponding to the class output from the class classification circuit 26, in a writingover manner. 
[0142] The prediction-tap memory 28 reads a stored value at the address corresponding to the class output from 

15 the class classification circuit 26, sends it to the adder 27, and stores the output of the adder 27 at the address. 

[0143] The adder 29 reads the master data serving as the focused pixel among the HD pixels constituting the HD 
image stored in the frame memory 21 ; the SD pixels sent from the predict ion -tap generation circuit 25; and a stored 
value at the address corresponding to the class output from the class classification circuit 26, that is, the multiplication 
results of the focused pixel and the SD pixels corresponding to the right side of each of the expressions (18), from the 

20 master data memory 30, and adds the stored value to the multiplication result of the focused pixel read from the frame 
memory 21 and the SD pixels sent from the predict ion -tap generation circuit 25 to perform the operation corresponding 
to the summation (L) at the right side of each of the normal equations (18). The adder 29 stores the operation result 
in the master-data memory 30 at the address corresponding to the class output from the class classification circuit 26, 
in a writing-over manner. 

25 [0144] In other words, the adders 27 and 29 perform multiplication shown in the expressions (18). Since the right 
side of each of the expressions (18) includes the multiplication of the master data "y" and an SD pixel "x," the SD pixel 
u x" is required for the multiplication performed by the adder 29, in addition to the master data "y." 
[0145] The master-data memory 30 reads a stored value at the address corresponding to the class output from the 
class classification circuit 26, sends it to the adder 29, and stores the output of the adder 29 at the address. 

30 [0146] The operation circuit 31 sequentially reads stored values at the address corresponding to each class from 
the prediction-tap memory 28 and the master-data memory 30, formulates the normal equations shown in the expres- 
sions (18), and solves the equations to obtain the prediction coefficients in each class. Specifically, the operation circuit 
31 uses the stored values at the address corresponding to each class from the prediction-tap memory 28 and the 
master-data memory 30 to formulate the normal equations shown in the expressions (18), solves the equations to 

35 obtain the prediction coefficients in each class, and sends them to the coefficient RAM 45. 

[0147] The processing of the data processing section 2 shown in Fig. 9 will be described next by referring to a 
flowchart shown in Fig. 11 . The operation of each circuit is controlled by a controller not shown. 
[0148] The data processing section 2 receives the SD image serving as the input data to be processed. The frame 
memory 41 sequentially stores the SD image serving as the input data to be processed, for example, in units of frames. 

40 [0149] In the data processing section 2, in a step S21, the class-tap generation circuit 42 and the prediction-tap 
generation circuit 43 read SD pixels disposed around an HD pixel for which a prediction value is to be obtained by 
tuning processing, from the frame memory 41 to configure a class tap and a prediction tap, respectively. The class tap 
and the prediction tap are sent to the class classification circuit 44 and the prediction operation circuit 46, respectively. 
[0150] When the class classification circuit 44 receives the class tap from the class-tap generation circuit 42, it obtains 

45 the class of the focused pixel according to the class tap and sends it to the coefficient RAM 45 as an address in a step 
S22. The coefficient RAM 45 reads prediction coefficients stored at the address corresponding to the class sent from 
the class classification circuit 44, and sends it to the prediction operation circuit 46 in a step S23. 
[0151] The prediction operation circuit 46 uses the prediction tap sent from the prediction -tap generation circuit 43 
and the prediction coefficients sent from the coefficient RAM 45 to perform the operation shown in the expression (12) 

so in a step S24. The prediction value E[y] of the focused pixel y is obtained and the processing proceeds to a step S25. 
In the step S25, the prediction-coefficient operation circuit 46 outputs the prediction value E[y] of the focused pixel "y n 
obtained in the step S24 as the pixel value of an HD pixel, and the processing proceeds to a step S26. 
[0152] In the step S26, it is determined whether all the HD pixels constituting the one-frame HD image corresponding 
to a one-frame SD image stored in the frame memory 41 have been processed as focused pixels. When it is determined 

55 that all the HD pixels have not yet been processed, the processing returns to the step S21 and the same processing 
is repeated with an HD pixel not yet serving as a focused pixel being handled as a focused pixel. 
[0153] On the other hand, in the step S26, when it is determined that all the HD pixels constituting the one-frame 
HD image have been processed as focused pixels, the processing is terminated. 
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[0154] The processing illustrated in the flowchart of Fig. 11 is performed, for example, every time a new SD image 
is stored in the frame memory 41 . 

[0155] Next, the processing of the real-time learning section 1 shown in Fig. 9 will be described by referring to a 
flowchart shown in Fig. 12. The operation of each circuit is controlled by a controller not shown. 

5 [0156] The real-time learning section 1 receives an HD image serving as input learning data. The frame memory 21 
sequentially stores the HD image serving as the input learning data, for example, in units of frames. 
[0157] In a step S31 , the thinning-out circuit 22 reads the HD image stored in the frame memory 21 and thins out 
pixels from the HD image to change it to an SD image. This SD image is sent to the frame memory 23 and stored in it. 
[0158] When the SD image is stored in the frame memory 23, the class-tap generation circuit 24 and the prediction- 

io tap generation circuit 25 use SD pixels constituting the SD image stored in the frame memory 23 to configure the class 
tap and the prediction tap of the focused pixel, a predetermined pixel among HD pixels constituting the HD image 
stored in the frame memory 21 , from SD pixels disposed around the focused pixel, respectively, and send to the class 
classification circuit 26,, the adder 27, and the adder 29 in a step S32. 

[0159] In a step S33, the class classification circuit 26 uses the class tap sent from the class-tap generation circuit 
is 24 to perform class classification, obtains the class corresponding to the focused pixel, and sends it to the prediction- 
tap memory 28 and to the master-data memory 30 as an address. 

[0160] The processing proceeds to a step S34, and the prediction tap and the master data are added to respective 
related values. 

[0161] Specifically, in the step S34, the prediction-tap memory 28 reads a stored value at the address corresponding 
20 to the class output from the class classification circuit 26 and sends it to the adder 27. The adder 27 adds the stored 
value sent from the prediction-tap memory 28 to the multiplication results of the predetermined SD pixels constituting 
the prediction tap sent from the prediction-tap generation circuit 25 to perform the operation corresponding to the 
summation (X) serving as the multiplier of the prediction coefficient at the left side of each of the normal equations (18). 
The adder 27 stores the operation result in the prediction-tap memory 28 at the address corresponding to the class 
25 output from the class classification circuit 26, in a writing-over manner 

[0162] In the step S34, the master-data memory 30 reads a stored value at the address corresponding to the class 
output from the class classification circuit 26 and sends it to the adder 29. The adder 29 reads the focused pixel among 
the HD pixels constituting the HD image stored in the frame memory 21 as master data (further reads the predetermined 
SD pixels constituting the prediction tap sent from the prediction-tap generation circuit 25); and adds the multiplication 
30 results of the master data and the predetermined SD pixels to the stored value sent from the master<lata memory 30 
to perform the operation corresponding to the summation (X) at the right side of each of the normal equations (18). 
The adder 29 stores the operation result in the master-data memory 30 at the address corresponding to the class 
output from the class classification circuit 26, in a writing-over manner. 

[0163] Then, the processing proceeds to a step S35, and it is determined whether all the HD pixels constituting the 
35 one-frame HD image stored in the frame memory 21 have been processed as focused pixels. When it is determined 
that all the HD pixels have not yet been processed, the processing returns to the step S32 and the same processing 
is repeated with an HD pixel not yet serving as a focused pixel being handled as a focused pixel. 
[0164] On the other hand, in the step S35, when it is determined that all the HD pixels constituting the one-frame 
HD image have been processed as focused pixels, the processing proceeds to a step S36. The operation circuit 31 
40 sequentially reads stored values at the address corresponding to each class from the prediction-tap memory 28 and 
the master-data memory 30, formulates the normal equations shown in the expressions (18), and solves the equations 
to obtain the prediction coefficients in each class. In addition, the operation circuit 31 sends the obtained prediction 
coefficients of each class to the coefficient RAM 45 and stores them in it, and the processing is terminated. 
[0165] With these operations : the prediction coefficients of each class stored in the coefficient RAM 45 are updated 
45 every time the processing shown in the flowchart of Fig. 12 is performed. 

[0166] The processing shown in the flowchart of Fig. 12 is performed, for example, every time a new HD image is 
stored in the frame mem^iy 21 . 

[01 67] In the learning processing of prediction coefficients performed in the real-time learning section 1 , as described 
above, when the required number of normal equations for obtaining prediction coefficients cannot be obtained in a 

50 class, prediction coefficients stored in the coefficient RAM 45 are, for example, not updated and the prediction coeffi- 
cients specified first is used as is in the processing of the data processing section 2 for the class. 
[0168] The processing shown in each of the flowcharts of Figs. 11 and 12 is performed in parallel. 
[0169] As described above, the storage contents of the prediction -tap memory 28 and the master-data memory 30 
are not reset; the prediction tap and the master data are added to respective related values; the normal equations (18) 

55 are formulated with the use of the addition results; and the prediction coefficients of each class are obtained in real 
time in the real-time learning section 1 . As a result, the prediction coefficients of each class become more appropriate 
for the SD image serving as the input data to be processed as learning processing is performed with the use of the 
HD image serving as input learning data. Therefore, the SD image is adaptiyely processed and the HD image obtained 
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by the processing is improved every moment as learning proceeds. More specifically, the resolution and other factors 
of the HD image to be output from the data processing section 2 are improved every moment. 

[0170] In the real-time learning section 1 , the prediction tap can be added to a related value in the adder 27 after at 
least one of the stored value sent from the prediction-tap memory 2B and the multiplication results of the predetermined 

5 SD pixels constituting the prediction tap sent from the prediction-tap generation circuit 25 is multiplied by a predeter- 
mined weight. When the adder 27 multiplies the stored value sent from the prediction-tap memory 28 by a weight 
smaller than the multiplication results of the predetermined SD pixels constituting the prediction tap sent from the 
prediction-tap generation circuit 25, the real-time learning section 1 can learn prediction coefficients while an HD image 
serving as past input learning data is ignored at a predetermined rate. When the adder 27 performs the above-described 

io weighting., the same weighting operation is necessary for the addition in the adder 29. 

[0171] In addition, when the above-described weight (ignoring) is employed, the real-time learning section 1 can be 
configured such that the S/N ratio is determined from a signal in the blanking period in TV broadcasting; and a weight 
for an obtained prediction tap is increased when the S/N ratio is high, and a weight for an obtained prediction tap is 
reduced when the S/N ratio is low. In this case, the real-time learning section 1 can perform learning with a noise effect 

1 $ being reduced. 

[0172] The real-time learning section 1 is configured such that a change in scene is detected and a weight applied 
to the stored value sent from the prediction-tap memory 28 is set to zero when the scene is changed. In this case, 
when the scene is changed, the real-time learning section 1 ignores all past learning and restarts learning from the 
beginning. Therefore, immediately after the scene is changed, the image quality of the HD image output from the data 
20 processing section 2 deteriorates, but the image quality of the HD image output from the data processing section 2 is 
improved every moment as learning in the real-time learning section 1 proceeds. 

[0173] In the above case, the SD image serving as the input data to be processed and the HD image serving as the 
input learning data having the same contents as the SD image are broadcasted together. The SD image serving as 
the input data to be processed and the HD image serving as the input learning data can be broadcasted at different 
25 timing. In this case, it is necessary for the image processing apparatus 200 shown in Fig. 9 to send the SD image 
serving as the input data to be processed to the data processing section 2 when it is received and to send the HD 
image serving as the input learning data to the real-time learning section 1 when it is received. 

[0174] Fig. 1 3 shows an example structure of an image processing apparatus 300 to which the self-tuning processing 
circuit shown in Fig. 2 is applied, according to another embodiment. The same symbols as those used in Fig. 9 are 

30 assigned to the portions corresponding to those in Fig. 9, and the descriptions thereof will be omitted, if unnecessary. 
[0175] Also in this image processing apparatus 300, class classification tuning processing is performed to convert 
an SD image to an HD image in the same way as in the image processing apparatus 200 shown in Fig. 9. In the image 
processing apparatus 200 shown in Fig. 9, the structures of a class tap and a prediction tap are fixed to four-by-four 
SD pixels around a focused pixel as shown in Fig. 10, and real-time learning is applied only to the prediction coefficients 

35 of each class. In the image processing apparatus shown in Fig. 13, however, real-time learning is applied not only to 
the prediction coefficients of each class but to the structures of a class tap and a prediction tap. 
[0176] Specifically, a real-time learning section 1 shown in Fig. 1 3 includes a frame memory 21 , a thinning-out circuit 
22, a frame memory 23, a feature-amount extraction circuit 51, a tap-candidate generation circuit 52, N learning 
processing sections 53-, to 53 N: and a determination circuit 54. 

ao [0177] The feature-amount extraction circuit 51 reads an SD image obtained by thinning out pixels from an HD image 
serving as input learning data and stored in a frame memory 23, extracts a feature amount of the SD image, and sends 
it to the tap-candidate generation circuit 52. 

[0178] More specifically, the feature-amount extraction circuit 51 , for example, calculates correlation of an SD pixel, 
a pixel to be processed, constituting the SD image stored in the frame memory 23 with an SD pixel disposed in the 
45 vicinity in terms of space and time of the pixel to be processed, one after another. The feature-amount extraction circuit 
51 obtains the average of the correlation of all SD pixels constituting the SD image stored in the frame memory 23 as 
a feature amount of the SD image, and sends it to the tap-candidate generation circuit 52. 

[0179] The tap-candidate generation circuit 52 generates the structures of a plurality of class taps and prediction 
taps as class-tap and prediction-tap candidates according to the feature amount sent from the feature-amount extrac- 
50 tion circuit 51. 

[0180] Specifically, the tap-candidate generation circuit 52 generates class-tap and prediction-tap candidates using 
many SD pixels disposed in the vicinity of the focused pixel when the SD image has a highly correlated feature amount. 
Hereinafter, a class tap and a prediction tap are collectively called a tap, if necessary. The tap-candidate generation 
circuit 52 generates tap candidates also using an SD pixel disposed relatively distant from the focused pixel when the 
55 SD image has a low correlated feature amount. More specifically, for example, when the SD image has a highly cor- 
related feature amount, tap candidates using SD pixels adjacent to each other are generated, and when the SD image 
has a low correlated feature amount, tap candidates using SD pixels disposed at an interval of some pixels (such as 
one pixel or two pixels) are generated. 
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[0181] A plurality of tap candidates are generated, for example, by preparing a standard tap structure for an SD- 
image feature amount and by widening or narrowing the interval of SD pixels in one or more directions among the 
horizontal direction, the vertical direction, and the time-domain direction. In the embodiment shown in Fig. 10, taps are 
configured with SD pixels disposed against the focused pixel in the space-domain direction. Taps can be configured 
with the use of SD pixels disposed against the focused pixel in the time-domain direction. 

[0182] In the embodiment shown in Fig. 13, N tap candidates are generated and each of the N tap candidates is 
sent to the corresponding section among the learning processing sections 53^ to 53 N in the tap-candidate generation 
circuit 52. More specifically, for example, when information indicating the structure of the n-th (n = 1, 2, N) tap 
candidate among the N tap candidates is called tap-structure information #n, the tap-structure information #n is sent 
to the learning processing section 53 n . 

[0183] The learning processing section 53 n receives the HD image and the SD image stored in the frame memory 
21 and the frame memory 23 as well as the tap-structure information #n. The learning processing section 53 n configures 
a class tap and a prediction tap having the structure corresponding to the tap-structure information #n, and obtains the 
prediction coefficients of each class in the same way as in the real-time learning section 1 shown in Fig. 9. The learning 
processing section 53 n applies class classification tuning processing to the SD image sent from the frame memory 23 
with the use of the prediction coefficients of each class to convert the SD image to an HD image. Then, the learning 
processing section 53n obtains a prediction error between the converted HD image and the HD image stored in the 
frame memory 21 and sends it to the determination circuit 54, together with the prediction coefficients of each class 
obtained by learning and the tap-structure information #n sent from the tap-candidate generation circuit 52. 
[0184] The determination circuit 54 detects the minimum value among the prediction errors sent from the learning 
processing sections 53^ to 53 N , sends to the coefficient RAM 45 the prediction coefficients of each class output from 
the learning processing section, the most appropriate learning processing section, which sends the minimum prediction 
error among the learning processing sections 53-, to 53 N , and sends the tap-structure information output from the most 
appropriate learning processing section to a class-tap generation circuit 42 and a prediction-tap generation circuit 43. 
[0185] Fig. 14 shows an example structure of the learning processing section 53 n shown in Fig. 13. 
[0186] The learning processing section 53 n is formed of a prediction-coefficient learning section 61 , a class-classi- 
fication tuning processing section 62, and a prediction-error calculation section 63. The prediction-coefficient learning 
section 61 receives the HD image from the frame memory 21 , the SD image from the frame memory 23, and the tap- 
structure information #n from the tap-candidate generation circuit 52. The class-classification tuning processing section 
62 receives the SD image from the frame memory 23, the tap-structure information #n from the tap-candidate generation 
circuit 52, and the prediction coefficients sent from the prediction-coefficient learning section 61. The prediction-error 
calculation section 63 receives the HD image from the frame memory 21 and the prediction value sent from the class- 
classification tuning processing section 62. 

[0187] The prediction-coefficient learning section 61 uses a class tap and a prediction tap having the structure cor- 
responding to the tap-structure information #n to obtain the prediction coefficients of each class by learning according 
to the HD image and the SD image in the same way as in the real-time learning section 1 shown in Fig. 9, and sends 
them to the class-classification tuning processing section 62. The prediction coefficients of each class obtained by the 
prediction-coefficient learning section 61 are also sent to the determination circuit 54. 

[0188] The class-classification tuning processing section 62 performs class-classification tuning processing with the 
use of the prediction coefficients of each class sent from the predict ion -coefficient learning section 61 in the same way 
as in the data processing section 2 in Fig. 9 to obtain the prediction value of an HD pixel from SD pixels constituting 
the SD image, and sends the prediction value to the prediction-error calculation section 63. 

[01 89] The prediction-error calculation section 63 obtains a prediction error such as the sum square of the differences 
between an HD pixel constituting the HD image stored in the frame memory 21 and the prediction value of the HD 
pixel sent from the class-classification tuning processing section 62 corresponding to the HD pixel in the HD image 
and sends it to the determination circuit 54. 

[0190] The learning processii »y section 53 n sends the tap-structure information #n sent from the tap-candidate gen- 
eration circuit 52 to the determination circuit 54, in addition to the prediction coefficients of each class and the prediction 
error obtained therein. 

[0191] Fig. 15 shows an example structure of the prediction-coefficient learning section 61 shown in Fig. 14. 
[0192] The prediction-coefficient learning section 61 is formed of a class-tap generation circuit 71, a prediction-tap 
generation circuit 72, a class classification circuit 73, an adder 74, a predict ion -tap memory 75, an adder 76, a master- 
data memory 77, and an operation circuit 78, which are configured in the same way as the class-tap generation circuit 
24, the prediction-tap generation circuit 25, the class classification circuit 26, the adder 27, the prediction-tap memory 
28, the adder 29, the master-data memory 30, and the operation circuit 31 shown in Fig. 9. 

[0193] The class-tap generation circuit 71 and the prediction-tap generation circuit 72 receive the SD image stored 
in the frame memory 23, and the adder 76 receives the HD image stored in the frame memory 21 and predetermined 
SD pixels sent from the prediction-tap generation circuit 72. The prediction coefficients ol each class are obtained in 
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the same way as in the real-time learning processing section 1 shown in Fig. 9. 

[0194] The class-tap generation circuit 71 and the prediction-tap generation circuit 72 also receive the tap-structure 
information #n output from the tap-candidate generation circuit 52. The class-tap generation circuit 71 and the predic- 
tion-tap generation circuit 72 configure a class tap and a prediction tap having the structure corresponding to the tap- 
5 structure information #n, respectively. 

[0195] In the present embodiment, the class tap has the same structure as the prediction tap. The class tap and the 
prediction tap can have different structures. 

[0196] Fig. 16 shows an example structure of the class-classification tuning processing section 62 shown in Fig. 14. 
[0197] The class-classification tuning processing section 62 is formed of a class-tap generation circuit 81 , a predic- 
ts tion-tap generation circuit 82, a class classification circuit 83, a coefficient RAM 84, and a prediction operation circuit 
85 which are configured in the same way as the class-tap generation circuit 42, the prediction-tap generation circuit 
43, the class classification circuit 44, the coefficient RAM 45, and the prediction operation circuit 46 shown in Fig. 9, 
respectively. 

[0198] The class-tap generation circuit 81 and the prediction-tap generation circuit 82 receive the SD image stored 
in the frame memory 23, and the coefficient RAM 84 receives the prediction coefficients of each class obtained by the 
operation circuit 78 of the prediction-coefficient learning section 61 shown in Fig. 1 5. The SD image is converted to an 
HD image with the use of the prediction coefficients of each class stored in the coefficient RAM 84 in the same way 
as in the data processing section 2 shown in Fig. 9. 

[0199] The class-tap generation circuit 81 and the predict ion -tap generation circuit 82 also receive the tap-structure 

20 information #n output from the tap-candidate generation circuit 52. The class-tap generation circuit 81 and the predic- 
tion-tap generation circuit 82 configure a class tap and a prediction tap having the structure corresponding to the tap- 
structure information #n, respectively, namely, the same structure as the class tap and the prediction tap which the 
class-tap generation circuit 71 and the prediction -tap generation circuit 72 shown in Fig. 15 configure, respectively. 
[0200] The processing of the real-time learning section 1 shown in Fig. 13 will be described next by referring to a 

25 flowchart shown in Fig. 17. The operation of each circuit is controlled by the controller not shown. 

[0201] The real-time learning section 1 receives an HD image serving as input learning data. The frame memory 21 
sequentially stores the HD image serving as the input learning data, for example, in units of frames. 
[0202] In a step S41 , the thinning-out circuit 22 reads the HD image stored in the frame memory 21 and thins out 
pixels from the HD image to change it to an SD image. This SD image is sent to the frame memory 23 and stored in it. 

30 [0203] When the SD image is stored in the frame memory 23, the feature -a mount extraction circuit 51 extracts the 
feature amount of the SD image sent from the frame memory 23 and sends it to the tap-candidate generation circuit 
52, in a step S42. In a step S43, the tap-candidate generation circuit 52 generates N tap candidates according to the 
feature amount sent from the feature-amount extraction circuit 51 and sends the tap-structure information #1 to #N 
indicating the structures of the generated tap candidates to the learning processing sections 53, to 53 N . 

35 [0204] The class-tap generation circuit 71 and the prediction-tap generation circuit 72 in the prediction-coefficient 
learning section 61 shown in Fig. 1 5 constituting the learning processing section 53 n shown in Fig. 1 4 configure a class 
tap and a prediction tap with a predetermined pixel among the HD pixels constituting the HD image stored in the frame 
memory 21 being set to the focused pixel, having the structure corresponding to the tap-structure information #n from 
the SD pixels stored in the frame memory 21, respectively, and send to the class classification circuit 73, the adder 

40 74, and the adder 76 in a step S44. 

[0205] In a step S45, the class classification circuit 73 uses the class tap sent from the class-tap generation circuit 
71 to perform class classification, obtains the class of the focused HD pixel, and sends it to the prediction-tap memory 
28 and to the master-data memory 30 as an address. 

[0206] On the other hand, the HD pixel serving as the focused pixel is read from the frame memory 21 and sent to 
45 the adder 76. 

[0207] The processing proceeds to a step S46, and the adders 74 and 76 perform additions to generate each term 
of the normal expression (18) related to the prediction tap and the master data, in the same way as in the step S34 
shown in Fig. 1 2. The addition results are sent to the prediction-tap memory 75 and the master-data memory 77, and 
stored in them at the address corresponding to the class output from the class classification circuit 73. 
50 [0208] Then, it is determined in a step S47 whether all the HD pixels constituting the one-frame HD image stored in 
the frame memory 21 have been processed as focused pixels. When it is determined that all the HD pixels have not 
yet been processed, the processing returns to the step S44 and the same processing is repeated with an HD pixel not 
yet serving as a focused pixel being handled as a focused pixel. 

[0209] On the other hand, in the step S47, when it is determined that all the HD pixels constituting the one^rame 
55 HD image have been processed as focused pixels, the processing proceeds to a step S48. The operation circuit 78 
sequentially reads stored values at the address corresponding to each class from the prediction-tap memory 28 and 
the master-data memory 30, formulates the normal equations shown in the expressions (18), and solves the equations 
to obtain the prediction coefficients in each class. In addition, the operation circuit 78 sends the obtained prediction 
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coefficients of each class to the coefficient RAM 84 of the class-classification tuning processing section 62 shown in 
Fig. 16 and stores them in it, namely, updates the storage contents of the coefficient RAM 84. 

[0210] The class-classification tuning processing section 62 performs class-classification tuning processing with the 
use of the prediction coefficients of each class stored in the coefficient RAM 84 to convert the SD image stored in the 

s frame memory 23 to an HD image (prediction values of the HD image stored in the frame memory 21 ) in a step S49. 
[0211] Specifically, in the class-classification tuning processing section 62 shown in Fig. 16, the class-tap generation 
circuit 81 and the prediction-tap generation circuit 82 configure a class tap and a generation tap having the structure 
corresponding to the tap-structure information #n with the use of the SD pixels sent from the frame memory 23, and 
sends them to the class classification circuit 83 and the prediction operation circuit 85, respectively. 

io [0212] The class classification circuit 83 obtains the class corresponding to the focused pixel in the HD image ac- 
cording to the class tap sent from the class-tap generation circuit 81, and sends it to the coefficient RAM 84 as an 
address. The coefficient RAM 84 reads prediction coefficients stored at the address corresponding to the class sent 
from the class classification circuit 83, and sends to the prediction operation circuit 85. 

[0213] The prediction operation circuit 85 performs the operation shown by the expression (12) with the use of the 
is prediction tap sent from the prediction-tap generation circuit 82 and the prediction coefficients sent from the coefficient 
RAM 84 to obtain the prediction value E[y] of the focused pixel "y" and sends it to the prediction-error calculation section 
63 shown in Fig. 14. 

[0214] In a step S50, the prediction -error calculation section 63 obtains a prediction error between the prediction 
value E[y] output from the class-classification tuning processing section 62 and the HD image stored in the frame 
20 memory 21 , and sends it to the determination circuit 54 shown in Fig. 1 3 together with the prediction coefficients of 
each class output from the prediction-coefficient learning section 61 and the tap-structure information #n. 
[0215] The determination circuit 54 detects the minimum value among the prediction errors sent from the learning 
processing sections 53 n to 53 N , in a step S51 ; and sends the prediction coefficients of each class output from the most 
appropriate learning processing section which has obtained the minimum prediction error among the learning process- 
es ing sections 53-, to 53 N to the coefficient RAM 45 of the data processing section 2, and the tap-structure information 
output from the most appropriate learning processing section to the class-tap generation circuit 42 and the prediction- 
tap generation circuit 43 of the data processing section 2 in a step S52. Then, the processing is terminated. 
[0216] The processing shown in the flowchart of Fig. 17 is performed, for example, every time a new HD image is 
stored in the frame memory 21 . 
30 [0217] In the data processing section 2 shown in Fig. 13, the storage contents of the coefficient RAM 45 are updated 
by the prediction coefficients ol each class sent from the determination circuit 54 of the real-time learning section 1. 
The class-tap generation circuit 42 and the prediction-tap generation circuit 43 in the data processing section 2 configu re 
a class tap and a prediction tap having the structure corresponding to the tap-structure information sent from the real- 
time learning section 1 , and the SD image is converted to an HD image in the same way as in Fig. 9. 
35 [0218] As described above, when the real-time learning section 1 learns the structure of a class tap and a prediction 
tap in real time in addition to the prediction coefficients of each class, an HD image having a further improved image 
quality is obtained. 

[021 9] Even when the real-time learning section 1 learns the structure of a class tap in real time, if the above-described 
weight is introduced in the step S46 of Fig. 17, where additions are made to the prediction tap and the master data of 
40 each class, the structure of the class tap can be learned while the HD image serving as past input learning data is 
ignored at a predetermined rate. 

[0220] When the structure of the class tap is learned in real time, since the structure of the class tap configured for 
the focused pixel is changed, the class of the focused pixel, obtained as a result of class classification performed 
according to the class tap, is also changed. 

45 [0221] More specifically, for example, to simplify a description, assuming that each SD pixel constituting a class tap 
is expressed by one bit, and the structure of a class tap is changed, for example, to one of two cases, a case in which 
the class tap is formed of adjacent foui -by-four SD pixels around the focused pixel, as shown in Fig. 10, and a case 
in which the class tap is formed of four-by-four SD pixels disposed every other pixels around the focused pixel, as 
shown in Fig. 18, the number of classes required is 2 16 + 2 16 . When the class tap is formed of four-by-four SD pixels 

so disposed every other pixels, the focused pixel is not disposed at the center of the SD pixels. The HD focused pixel can 
be, however, predicted when the real-time learning section 1 learns with the same phase relationship. 
[0222] In other words, when the class tap is either formed of the adjacent four-by-four SD pixels around the focused 
pixel or formed of the four-by-four SD pixels disposed every other pixels around the focused pixel, the number of 
classes required is 2 16 in either case. In contrast, when the class tap can be formed of the adjacent four-by-four SD 

55 pixels around the focused pixel and it can also be formed of the four-by-four SD pixels disposed every other pixels 
around the focused pixel, the number of classes required is 2 16 for a case in which the class tap is formed of the 
adjacent four-by-four SD pixels around the focused pixel, plus 2 16 for a case in which the class tap is formed of the 
four-by-four SD pixels disposed every other pixels around the focused pixel. 
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[0223] Since 2 16 classes required when the class tap is formed of the adjacent four-by-four SD pixels around the 
focused pixel are different from 2 16 classes required when the class tap is formed of the four-by-four SD pixels disposed 
every other pixels around the focused pixel, when the structure of the class tap is changed, how to perform the addition 
of the prediction tap in each class and the addition of the master data in each class in the step S46 in Fig. 17, for 
5 example, becomes an issue. This can be performed, for example, in the following way. 

[0224] Specifically, the number of classes which matches the total number of classes required for each structure of 
a class tap to be employed by real-time learning are prepared in advance, and when a class tap having a certain 
structure is employed, classes other than the classes assigned to the class tap having the structure are not used, 
namely, are ignored. 

io [0225] More specifically, when class taps to be employed by real-time learning are, for example, a first class tap 
formed of 1 0-by-1 0 SD pixels around the focused pixel and a second class tap formed of four-by-four SD pixels around 
the focused pixel, as shown in Fig. 18, since 2 100 and 2 16 classes are required for the first and second class taps, 
respectively, 2 100 + 2 16 classes are prepared where each SD pixel is expressed by one bit. Among the 2 100 + 2 16 
classes, 2 100 and 2 16 classes are assigned to the first and second class taps, respectively. When the first class tap is 

is employed, processing is performed only for the 2 100 classes assigned thereto. When the second class tap is employed, 
processing is performed only for the 2 16 classes assigned thereto. 

[0226] When the structure of a prediction tap is changed, the prediction coefficients n w" to be multiplied by SD pixels 
"x" constituting the prediction tap are also changed in the expression (1 2). The prediction coefficients in the expressions 
(1 8) are also changed. To handle this issue, prediction coefficients are prepared for all SD pixels which can be prediction 
20 taps for the focused pixel, and the operation is performed with the pixel values of SD pixels which are not employed 
as prediction taps being set to zero among the pixel values of the SD pixels serving as multipliers in the expression 
(12) for the prediction coefficients. 

[0227] It is also possible that the structure of a class tap is fixed and classes required for the fixed class tap are 
streamlined by the real-time learning section 1 . Specifically, for example, when a class tap is formed of "A" SD pixels 

25 and each SD pixel constituting the class tap is expressed by U B M bits, (2 A ) B classes are usually required. The image 
quality of an HD image output from the data processing section 2 is improved in some cases when the real-time learning 
section 1 eliminates a class or integrates a class and another class into one class. In other words, the same advantages 
as in a case when the structure of the class tap is changed are achieved in some cases when the real-time learning 
section 1 streamlines classes. It is also possible that, for example, the real-time learning section 1 learns class-stream- 

30 lining in real time such that the above-described prediction error becomes small. 

[0228] In the above case, the data processing section 2 performs linear processing. The present invention can also 
be applied to a case when the data processing section 2 performs non-linear processing. 

[0229] In the embodiment shown in Fig. 13, the learning processing section 53 n uses a class tap and a prediction 
tap having the structure corresponding tothe tap-structure information #n sent from the tap-candidate generation circuit 

35 52 as is to obtain prediction coefficients until new tap-structure information #n is received. The learning processing 
section 53 n can be configured such that the feature amount of a focused pixel, such as the correlation of SD pixels 
disposed around the focused pixel, is obtained, the class tap and the prediction tap having the structure corresponding 
to the tap-structure information #n is changed for each focused pixel to have a structure based on the feature amount, 
and prediction coefficients are obtained. 

40 [0230] In this case, for example, it is necessary to prepare many classes correspondingly to the number of variations 
to which the class tap and the prediction tap having the structure corresponding to the tap-structure information #n are 
modified. In the data processing section 2, in the same way as in the above-described case, it is necessary that the 
feature amount of a focused pixel is obtained, and the class-tap generation circuit 42 and the prediction-tap generation 
circuit 43 configure a class tap and a prediction tap having a structure obtained by modifying according to the feature 

45 amount the structure corresponding the tap-structure information #n sent from the determination circuit 54. The feature 
amount can be anything indicating a feature of SD pixels around the focused pixel, such as the DR, dispersion, or 
standard deviation of the SD pixels. 

[0231] Embodiments of the present invention have been described by the circuits shown in Fig. 9 and Fig. 13. Soft- 
ware which implements the function of each circuit by a program can be used. 
so [0232] The series of processing shown in Fig. 8, Fig. 11, Fig. 12, and Fig. 17 can be implemented by hardware or 
software. When the series of processing is achieved by software, a program constituting the software is installed into 
a computer built in the NR processing apparatus 100 and the image processing apparatuses 200 and 300 serving as 
special hardware or into a general-purpose computer. 

[0233] Recording media which store the program for executing the series of processing described above will be 
55 described below by referring to Figs. 19 A, 19B, and 1 9C. The program is installed into a computer and is made exe- 
cutable by the computer. 

[0234] The program is recorded in advance into a hard disk 102 or a semiconductor memory 103 serving as a re- 
cording medium which is built in a computer 101, as shown in Fig. 19A. 
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[0235] Alternatively, the program is recorded temporarily or perpetually into recording media, such as a floppy disk 
111, a compact disc read-only memory (CD-ROM) 112, a magnetooptical (MO) disk 113, a digital versatile disk (DVD) 
114, a magnetic disk 115 : and a semiconductor memory 116, as shown in Fig. 19B. 

[0236] The program is installed in the computer from the above-described recording medium. Alternatively, the pro- 
gram is transferred by radio from a downloading site 121 to the computer 101 through an artificial satellite for digital 
satellite broadcasting, or to the computer 101 by wire through a network 131 such as a local area network (LAN) or 
the Internet; and is installed into the hard disk 102, built in the computer 101, as shown in Fig. 19C. 
[0237] In the present specification, the steps describing the programs for executing various types of processing are 
not necessarily processed in a time sequential manner in the order in which the steps are described in the flowcharts. 
Processing to be executed in parallel or individually, such as parallel processing or processing by objects, is also 
included. 

[0238] Fig. 20 shows an example structure of the computer 101 shown in Fig. 1 9A. 

[0239] The computer 101 includes a central processing unit (CPU) 142, as shown in Fig. 20. The CPU 142 is con- 
nected to an input and output interface 145 through a bus 1 41 . When the user operates an input section 1 47 formed 
of a keyboard and a mouse to input a command through the input and output interface 145, the CPU 142 executes a 
program stored in a read-only memory (ROM) 143 corresponding to the semiconductor memory 103 shown in Fig. 
19A. Alternatively, the CPU 142 loads into a random access memory (RAM) 144 a program stored in the hard disk 
102; a program transferred through the satellite 122 or the network 131, received by a communication section 148, 
and installed into the hard disk 102; or a program read from the floppy disk 111 , the CD-ROM 112. the MO disk 113, 
the DVD 114, or the magnetic disk 115 which is loaded into a drive 149 : and installed into the hard disk 102; and 
executes it. The CPU 142 outputs the processing result, for example, through the input and output interface 145 to a 
display section 146 formed of a liquid-crystal display (LCD), as required. 



25 Claims 

1. A data processing apparatus for processing input data and outputting the processed data, comprising: 

data processing means for processing the input data by a predetermined processing method and outputting 
30 the processed data; and 

real-time learning means lor controlling such that the processing method is learned in real time and said data 
processing means processes the input data by the learned processing method, so that the output data is 
improved as time elapses. 

35 2. A data processing apparatus according to Claim 1, wherein said real-time learning means learns the processing 
method according to the input data or the output data. 

3. A data processing apparatus according to claim 1, wherein said real-time learning means learns the processing 
method according to special data for learning the processing method. 

40 

4. A data processing apparatus according to Claim 1, wherein said real-time learning means learns the processing 
method according to time-sequentially input learning data. 

5. A data processing apparatus according to Claim 4, wherein said real-time learning means learns the processing 
45 method according to data having a value within a predetermined range from the current value among the time- 
sequentially input learning data. 

6. A data processing apparatus according to Claim 4, wherein said real-time learning means learns the processing 
method while said real-time learning means ignores past learning data at a predetermined rate. 



so 



7. A data processing method for processing input data and outputting the processed data, comprising: 



a data processing step for processing the input data by a predetermined processing method and outputting 
the processed data; and 

55 a real-time learning step for controlling such that the processing method is learned in real time and the input 

data is processed by the learned processing method in said data processing step, so that the output data is 
improved as time elapses. 
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8. A data processing apparatus for processing input data and outputting the processed data, comprising: 

data processing means for processing the input data by a linear coupling of the input data and a predetermined 
prediction coefficient and outputting the processed data; and 

real-time learning means for learning the prediction coefficient so that the output data has higher quality as 
time elapses and for transferring the prediction coefficient to said data processing means. 

9. A data processing apparatus according to claim 8, wherein said real-time learning means learns the prediction 
coefficient according to special data for learning the prediction coefficient. 

10. A data processing apparatus according to Claim 8, wherein said real-time learning means learns the prediction 
coefficient according to time-sequentially input learning data. 

11. A data processing apparatus according to Claim 10, wherein said real-time learning means learns the prediction 
is coefficient while said real-time learning means ignores past learning data at a predetermined rate. 

12. A data processing method for processing input data and outputting the processed data, comprising: 

a data processing step for processing the input data by a linear coupling of the input data and a predetermined 
20 prediction coefficient and outputting the processed data; and 

a real-time learning step for learning the prediction coefficient so that the output data has higher quality as 
time elapses and for transferring the prediction coefficient to said data processing step. 

13. A storage medium for storing a computer-controllable program for processing input data and outputting the proc- 
25 essed data, the program comprising: 

a data processing step for processing the input data by a predetermined processing method and outputting 
the processed data; and 

a real-time learning step for controlling such that the processing method is learned in real time and the input 
30 data is processed by the learned processing method in said data processing step, so that the output data is 

improved as time elapses. 

. 14. A storage medium for storing a computer-controllable program for processing input data and outputting the proc- 
essed data, the program comprising: 

35 

a data processing step for processing the input data by a linear coupling of the input data and a predetermined 
prediction coefficient and outputting the processed data; and 

a real-time learning step for learning the prediction coefficient so that the output data has higher quality as 
time elapses and for transferring the prediction coefficient to said data processing step. 

15. A computer program for processing input data and outputting the processed data, the program comprising: 

a data processing step for processing the input data by a predetermined processing method and outputting 
the processed data; and 

45 a real-time learning step for controlling such that the processing method is learned in real time and the input 

data is processed by the learned processing method in said data processing step, so that the output data is 
improved as time elapses. 



40 



50 



55 



16. A computer program for processing input data and outputting the processed data, the program comprising: 

a data processing step for processing the input data by a linear coupling of the input data and a predetermined 
prediction coefficient and outputting the processed data; and 

a real-time learning step for learning the prediction coefficient so that the output data has higher quality as 
time elapses and for transferring the prediction coefficient to said data processing step. 
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FIG. 2 
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FIG. 3A 



FIG. 3B 
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FIG. 4A 



FIG. 4B 
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FIG. 5 
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FIG. 6 
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FIG. 7A 




FIG. 7B 
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FIG. 8 
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FIG. 10 
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FIG. 11 
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FIG. 12 



PROCESSING OF REAL- 
TIME LEARNING SECTION 



THIN OUT PIXELS 



CONFIGURE CLASS TAP AND 
PREDICTION TAP 



T 



PERFORM CLASS CLASSIFICATION 



ADD PREDICTION TAP AND 
MASTER DATA TO RESPECTIVE 
RELATED VALUES 



NO 



S31 

S32 

S33 
S34 



/ — v 

\ HAS ONE FRAME BEEN FINISHED? } 



S35 



1 


YES 

f 


FORMULATE NORMAL EQUATION AND 
OBTAIN PREDICTION COEFFICIENT 




r 


TRANSFER PREDICTION COEFFICIENT 


> 


r 



S36 



S37 



c 



END 



J 



BN5DOCID: <EP 1001371A1_I_> 



33 



EP 1 001 371 A1 



UJ 
CD 
< 





— > 


CTION 
IT 


oocc 




PREDI 
OPERj 
OIHCU 



CO 



CD 




34 



BNSDOCID: <EP 1O01371A1_l_> 



EP 1 001 371 A1 



uj c 
en 



Olu 
^iZ 

QLJ- 
LU LU 
COO 
CLO 



CO 



CD 



o — 



o 

Q 



ISO 

— — <: o 
ac O uj uj 



UJ LU 



S£ cc 

Q O 
UJ GC 

ac az 

CL LU 

A 



o9 

II 

CO QC 

2^ 



2: o 
o o o i — 

LU CC JO 

GCOC <LU 

a. uj o co 



CO 
CO 



o 

— - LU 

o ^ 

ujZj 
az<: 



O LU 

LU UJ 
CCO 
Q-O 
> 



9 CO 

COGO = S^O 

— I -J ID CC Sr 
OOl— Q-O 



. CM 



LU LU C 

co _ u2 

CO QC 



35 

BNSDOCID: <EP 1001 371 A1_I_> 



EP 1 001 371 A1 



O uj 
i — o 
S^d 

Q LU 
LU UJ 
CO 

CL O 

t 



*3 

OO 



O 
Li- 



en. 



_ O 



o 

I— >- 

O QC 

5 o 

UJcl.2 
cr <r UJ 



<: 
o 



CO CO ^3 
CO CO n 

OOO 



LU 

o 

Q 



si 

co ac 



Dl2 
< h- 

co cc =5 

3 See 

OOO 



1 




QC 
UJ 

Q 
Q 

<: 



co 



§ 2 

Q Lug 
CL ^ 5? 

a.?oo 



CM 



LU 

O 

CO ^ 



LU 

CD 



36 



BNSDOCID: <EP. 



,1001371A1_I_> 



EP 1 001 371 A1 



LU 

CD 



CO 



O uj 
I — o 

UJUJ 

tro 



o <c 

OCT 



CO, 
CO 



t 



Q CC 
LU UJ 

CC CL!± 
O-OO 



o 



CO 



O 

o 



coco 

CO CO o 



UJ C 

CO cc 



CO 



1 



Q-9 

co =5 

ooo 



o 5=> 



5>- 



o 

UJ q. z rr 
Q-hOO 



t 



.CVJ 
CO 



ULI 

o 

CO ^ 



BNSDOCID: <EP 1001371A1_L> 



37 



EP 1 001 371 A1 



FIG. 17 
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FIG. 18 



SD PIXEL 

s 



FIRST CLASS 



O O O O o o o 



HD PIXEL 



TAP 

SECOND CLASS TAP 

O;' O O O O 



o 




o 


o 


o 


o 


o 


o 


oi 


o 


o 


! ° 


o 


! o 


o 


o 


o 


o 


o 


o 


t 

o : 

1 


o 


o 


o 


o 


' o 


o 


• 


o 


• 


o 


• 


t 


• 


o 


o 


o 


! ° 


o 


o 


o 


o 


o 


o 




o 


o 


! o 


o 


! ° 


o 


# 


o 


• 


o 


• 


J o 


• 


o 


o 


o" 


! o 


o 


o 


o 




o 


o 


! ° 


o 


o 


! o 


o 


! o 


o 


• 


o 


• 


o 


• 


! o 


• 


o 


o 


o 


! o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


! ° 


o 


I o 


o 


• 


o 


• 


o 


• 


o 


• 


o 


! ° 


o 


I ° 


o 


o 


o 


o 


o 


o 


o 


o 


o 


! o 


o 


[ o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


! o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 



o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 



ooooooooooooo 
ooooooooooooo 



o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 



o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 
o 



o o 
o o 



39 

BNSDOCID: <EP 1001371A1_I_> 



EP 1 001 371 A1 



FIG. 19A 
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FIG. 20 
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